configure C API CI aligned with runtime bindings - #360
Closed
ethanglaser wants to merge 12 commits into
Closed
Conversation
This pull request enhances the flexibility and predictability of blocked data structures by introducing explicit control over the number of elements per block (`blocksize_elements`) in addition to the existing byte-based blocking (`blocksize_bytes`). It also improves test coverage to verify the new behavior and edge cases. **Blocking parameter improvements:** * Added an optional `blocksize_elements` field to the `BlockingParameters` struct, allowing users to specify the number of elements per block directly. If set, this takes precedence over `blocksize_bytes` when determining block size. (`include/svs/core/data/simple.h`) **Testing and validation:** * Added test to demonstrate the pitfalls of relying solely on `blocksize_bytes` for memory prediction. * Extended unit tests to cover scenarios where `blocksize_elements` is set, including checks for correct block size selection and memory consumption predictions. (`tests/svs/core/data/block.cpp`)
Latest macos github actions image was bumped and no longer supports specified clang versions. This is because we take latest: https://github.com/intel/ScalableVectorSearch/blob/main/.github/workflows/build-macos.yaml#L35 and as far as I can tell there is not an easy way to track this via dependabot. Alternative would be to pin it to a version but then it could sit idle and forgotten.
This pull request refactors the `Blocked` class template to inherit from its allocator type instead of storing it as a member, and updates the associated test to use a struct-based allocator. This change simplifies allocator handling and improves compatibility with standard allocator patterns. **Core class refactoring:** * `Blocked` now inherits from its allocator type (`Alloc`) instead of storing an `Alloc allocator_` member, which simplifies construction, copying, and access to the allocator. The `get_allocator()` method now returns `*this` (as an allocator), and constructors have been updated accordingly. **Test improvements:** * The test for `Blocked` with an allocator has been updated to use a struct-based allocator (`I`), which provides a `value_type` and integer value for testing propagation and compatibility with the new inheritance-based implementation.
- Introduced `svs_id_filter_interface` to define filtering operations. - Implemented `svs_index_search_topK` to support an optional ID filter for search operations. - Updated existing search functions to use the new filtered search capabilities. - Added a new source file `filtered_search.hpp` containing the logic for filtered top-K search. - Modified existing samples and tests to demonstrate and validate the new filtering functionality. - Marked the previous `svs_index_search` function as deprecated, directing users to use `svs_index_search_topK` instead.
## Summary
Adds `get_memory_usage()` returning the total number of bytes a Vamana
index has **allocated** — graph storage + vector data + metadata — so an
integrator can accurately report and bound SVS memory consumption.
Both the static `VamanaIndex` and the dynamic `MutableVamanaIndex` are
covered, and the method is plumbed through the orchestrator layers
(`VamanaInterface` virtual → `VamanaImpl` override → `Vamana` /
`DynamicVamana`) so it is callable on `svs::Vamana` and
`svs::DynamicVamana`.
## Why
Integrators (e.g. memory-bounded module hosts) need to account for
memory SVS allocates via `mmap`/blocked allocators, which bypass the
host's `malloc` accounting. The existing `blocksize_bytes()` reports
only the initial block size and is neither an upper nor lower bound on
the real footprint. `get_memory_usage()` reports the true allocated
total across all blocks plus metadata.
## What
Accounting is **capacity-based** (the bytes the containers have
reserved, not just live elements) so that block over-allocation is
reflected:
| Component | Source |
|---|---|
| `graph_bytes` | `graph_.get_data().capacity() * element_size()` |
| `data_bytes` | `data_.capacity() * data_.element_size()` |
| `metadata_bytes` (dynamic only) | slot-status vector + entry-point
list + estimate of the external/internal ID translation maps |
A `VamanaMemoryUsage { graph_bytes, data_bytes, metadata_bytes, total()
}` struct and `get_memory_breakdown()` expose the per-component split;
`get_memory_usage()` returns `get_memory_breakdown().total()`.
Notes:
- A `detail::dataset_allocated_bytes()` helper uses capacity-based
accounting when the dataset exposes `capacity()` (flat/blocked
`SimpleData`), and falls back to live element count otherwise (e.g.
`SQDataset`). No public accessors or signatures were changed.
- The ID-translation map byte size is not directly queryable; it is
estimated from the entry count (accurate to within a few percent), with
a comment noting the approximation.
## Tests
New unit tests at both the core-index and orchestrator levels for the
static and dynamic indices:
- `tests/svs/index/vamana/index.cpp`,
`tests/svs/index/vamana/dynamic_index.cpp`
- `tests/svs/orchestrators/vamana.cpp`,
`tests/svs/orchestrators/dynamic_vamana.cpp`
Assertions: usage `> 0` for a built index, breakdown components sum to
the total, and `graph_bytes`/`data_bytes` are non-zero. `[managers]` and
the touched index tags pass with no regressions.
…n) (#354) ## Summary Exposes memory accounting in the **C API** for the Valkey-search integration: - `svs_index_get_memory_usage(index, size_t* out_bytes, err)` — total allocated bytes. - `svs_index_get_memory_breakdown(index, svs_memory_breakdown_t* out, err)` — `{graph_bytes, data_bytes, metadata_bytes}` component split. ~~- `svs_index_element_size(index, size_t* out_bytes, err)` — bytes per stored vector.~~ (keep at data level) All follow the existing C API conventions (out-param + `svs_error_h`, `wrap_exceptions`), matching the Phase-A design in the memory-accounting contract (intel-innersource #333). ## Layers - **C API** (`bindings/c`): the three functions + `svs_memory_breakdown_t` in `svs_c.h`; interface virtuals + concrete overrides in `src/index.hpp`; impls in `src/svs_c.cpp`. - **Core / orchestrator**: brings in `get_memory_breakdown()` (`MemoryBreakdown` struct + capacity-based `svs::data::detail::dataset_allocated_bytes` helper) on `VamanaIndex` / `MutableVamanaIndex` and through the orchestrator, plus an `element_size()` accessor parallel to `dimensions()`. This mirrors the approved public PR #345 so the C API can build and test standalone; once #345 lands on `dev/c-api`, this reduces to just the C API layer. ## Tests `bindings/c/tests/c_api_index.cpp` (static) and `c_api_dynamic_index.cpp` (dynamic): usage > 0, breakdown total == usage, `graph_bytes`/`data_bytes` > 0 (metadata > 0 for dynamic), `element_size == sizeof(float) * dimensions`, and null-arg handling. Both test cases pass (84 / 166 assertions). Related: builds on #345; memory-accounting contract in intel-innersource #333 / #326.
Bumps the pinned LTO prebuilt SVS library (`bindings/cpp/CMakeLists.txt`) from `svs-shared-library-lto-nightly-2026-05-21-1429` to `svs-shared-library-lto-nightly-2026-07-21-127`, which includes `get_memory_usage()` / `get_memory_breakdown()` (#345). ## Why The `Build and unit tests for C++ runtime bindings (with static library, ON)` job links the runtime bindings against this prebuilt lib. The previously-pinned nightly predated #345, so it failed: ``` bindings/cpp/src/dynamic_vamana_index_impl.h:73: error: no member named 'get_memory_breakdown' in 'svs::DynamicVamana' ``` The new nightly was built from `main` (post-#345, via the private-repo submodule bump intel-innersource#338) and ships the method — verified the tarball's headers contain `get_memory_breakdown`. This should turn that CI job green. Follows the pattern of #311 (Update SVS_URL in binaries).
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the C API build/packaging and CI to more closely mirror the runtime bindings setup, with explicit LVQ/LeanVec expectations and an integration-style consumer build to validate the exported CMake package and symbol surface.
Changes:
- Add
SVS_REQUIRE_LTO_ARCHIVEand switch LVQ/LeanVec prebuilt downloads to thev0.4.0release artifacts, with a CMake-version guard forDOWNLOAD_EXTRACT_TIMESTAMP. - Tighten the exported C API target so pure-C consumers don’t inherit
OpenMP::OpenMP_CXXor acxx_std_20requirement. - Add C API consumer/integration tests and update the GitHub Actions workflow to build/test/package in the manylinux container, plus stricter LVQ/LeanVec test expectations.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| bindings/cpp/CMakeLists.txt | Adds LTO-archive requirement toggle and uses release tarballs with CMake-version-safe FetchContent args. |
| bindings/c/CMakeLists.txt | Prevents leaking C++/OpenMP requirements to consumers; aligns LVQ/LeanVec prebuilt handling with runtime bindings. |
| bindings/c/tests/consumer/main.c | New C-only consumer smoke test for the installed C API package. |
| bindings/c/tests/consumer/CMakeLists.txt | Standalone consumer project using find_package(svs_c_api) and linking svs::svs_c_api. |
| bindings/c/tests/CMakeLists.txt | Adds compile-time expectation flag for LVQ/LeanVec tests based on build config. |
| bindings/c/tests/c_api_test_utils.h | Makes compressed-storage assertions configuration-aware; adds storage_usable(). |
| bindings/c/tests/c_api_storage.cpp | Requires SQ storage to always succeed (public build invariant). |
| bindings/c/tests/c_api_index.cpp | Skips build/search for compressed storage when unusable; requires SQ always works. |
| bindings/c/samples/simple.c | Falls back to simple storage when LeanVec/LVQ are unavailable. |
| bindings/c/samples/save_load.c | Same fallback behavior for save/load sample. |
| bindings/c/samples/dynamic.c | Same fallback behavior for dynamic sample. |
| .github/workflows/build-cpp-runtime-bindings.yml | Passes REQUIRE_LTO_ARCHIVE into the runtime bindings container build. |
| .github/workflows/build-c-api-bindings.yml | Reworks C API CI to container build + artifact packaging + integration tests. |
| .github/scripts/test-c-api-unit.sh | Runs unit tests and executes samples in the container. |
| .github/scripts/test-c-api-bindings.sh | Integration test: validates package contents, exported symbols, and consumer build/run. |
| .github/scripts/build-cpp-runtime-bindings.sh | Propagates SVS_REQUIRE_LTO_ARCHIVE into the CMake configure. |
| .github/scripts/build-c-api-bindings.sh | New script to configure/build/install/package C API bindings with LVQ/LeanVec toggles. |
rfsaliev
reviewed
Aug 7, 2026
rfsaliev
left a comment
Member
There was a problem hiding this comment.
Looks good except:
error handle can be reused, so constructions like:
svs_error_free(error);
error = svs_error_create();
should be removed
…i-ci # Conflicts: # bindings/cpp/CMakeLists.txt
Merged
Member
Author
|
Thanks for reviews. With rebasing to main and C API branches and original C API CI branch there is a lot going on. Closing this in favor of #362 which adds these changes directly to C API branch |
ethanglaser
added a commit
that referenced
this pull request
Aug 11, 2026
#360 reopened directly to C API branch --------- Co-authored-by: Rafik Saliev <rafik.f.saliev@intel.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.